Systems and methods for optical material characterization of waste materials using machine learning

ABSTRACT

Systems and methods for optical material characterization of waste materials using machine learning are provided. In one embodiment, a system comprises: an imaging device configured to generate image frames an area and target objects within the area; an object characterization processor coupled to the imaging device and comprising Neural Processing Units and a Neural Network Parameter Set. The Neural Network Parameter Set stores learned parameters utilized by the one or more Neural Processing Units for characterizing the one or more target objects. The Neural Processing Units are configured by the Neural Network Parameter Set to detect a presence of a plurality of different materials within the image frames based on a plurality of different features. For a first image frame of the plurality of image frames, the Neural Processing Units outputs material characterization data that identifies which of the plurality of different materials are detected in the first image frame.

CROSS-REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of U.S. patentapplication Ser. No. 16/177,137, entitled SYSTEMS AND METHODS FOROPTICAL MATERIAL CHARACTERIZATION OF WASTE MATERIALS USING MACHINELEARNING filed Oct. 31, 2018, which claims priority to, and the benefitof, U.S. Provisional Patent Application No. 62/580,720 titled “SYSTEMSAND METHODS FOR OPTICAL MATERIAL CHARACTERIZATION OF WASTE MATERIALSUSING MACHINE LEARNING” filed on Nov. 2, 2017, each of which isincorporated herein by reference in its entirety.

BACKGROUND

Within many industrial facilities, objects are transported on conveyorbelts from one location to another. Often a conveyor belt will carry anunsorted mixture of various objects and materials. In some instances,like within recycling and waste management facilities for example, someof the objects may be considered desirable (e.g. valuable) materialswhile others may be considered undesirable contaminants. For example,the random and unsorted contents of a collection truck may be unloadedat the facility onto a conveyor belt. At that point the facilityoperator does not really know specific details about the types ofmaterial that have just been received. The facility operator maytherefore wish to be able to identify what material is being carried onthe conveyor belt in order to gather data about the type of materialbeing conveyed, and/or to identify target material for removal from theconveyor belts such as by a sorting robot. Although sorting personnelmay be stationed to manually sort and catalog materials as it istransported on the belt, the use of sorting personnel is limitingbecause they can vary in their speed, accuracy and efficiency and cansuffer from fatigue over the period of a shift.

For the reasons stated above and for other reasons stated below whichwill become apparent to those skilled in the art upon reading andunderstanding the specification, there is a need in the art for systemsand methods for optical material characterization of waste materialsusing machine learning.

SUMMARY

The embodiments of the present disclosure provide methods and systemsfor optical material characterization of waste materials using machinelearning and will be understood by reading and studying the followingspecification.

In one embodiment, an optical material characterization systemcomprises: at least one imaging device configured to generate imageframes that capture an image of an area and one or more target objectswithin the area; an object characterization processor coupled to the atleast one imaging device and comprising one or more Neural ProcessingUnits and a Neural Network Parameter Set, wherein the Neural NetworkParameter Set stores learned parameters utilized by the one or moreNeural Processing Units for characterizing the one or more targetobjects; wherein the one or more Neural Processing Units are configuredby the Neural Network Parameter Set to detect a presence of a pluralityof different materials within the image frames based on a plurality ofdifferent features; and wherein for a first image frame of the pluralityof image frames, the one or more Neural Processing Units outputsmaterial characterization data that identifies which of the plurality ofdifferent materials are detected in the first image frame.

DRAWINGS

Embodiments of the present disclosure can be more easily understood andfurther advantages and uses thereof more readily apparent, whenconsidered in view of the description of the preferred embodiments andthe following figures in which:

FIG. 1 is a diagram illustrating an optical material characterizationsystem of one embodiment of the present disclosure;

FIG. 2 is a diagram illustrating operation of neural processing units ofan object characterization processor of one embodiment of the presentdisclosure;

FIG. 3 is a diagram illustrating an example structure for one embodimentof a neural network of the neural processing units;

FIG. 4 is a diagram of a Fully Convolutional Neural Network of oneembodiment of the present disclosure;

FIG. 5 is a diagram illustrating an example display of materialcharacterization data at a user interface device for one embodiment ofthe present disclosure;

FIGS. 6, 7 and 7A is a diagram illustrating another optical materialcharacterization system of one embodiment of the present disclosure;

FIG. 8 is a diagram illustrating a process for differentiating newlycollected materials from previously collected materials for oneembodiment of the present disclosure; and

FIG. 9 is a diagram illustrating another example display of materialcharacterization data at a user interface device for one embodiment ofthe present disclosure.

In accordance with common practice, the various described features arenot drawn to scale but are drawn to emphasize features relevant to thepresent disclosure. Reference characters denote like elements throughoutfigures and text.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings that form a part hereof, and in which is shown byway of specific illustrative embodiments in which the embodiments may bepracticed. These embodiments are described in sufficient detail toenable those skilled in the art to practice the embodiments, and it isto be understood that other embodiments may be utilized and thatlogical, mechanical and electrical changes may be made without departingfrom the scope of the present disclosure. The following detaileddescription is, therefore, not to be taken in a limiting sense.

In many industrial facilities such as recycling and/or other wastesorting facilities, materials are often transported via conveyer beltsas various sorting machinery segregates the waste materials based onvarious criteria such as material type. For example, an initial materialintake conveyer belt may transport a collection of mixed wasteincluding, for example, glass, plastics and cardboard objects. Sortingmachinery, as the waste material is conveyed, may be used to selectivelysort and transfer different objects to different parts of the facilitybased on their material type. For example, glass bottles may besegregated and transported to a different part of the facility thatplastic milk containers. Although waste products travelling on aconveyer belt are used as example target objects in the exampleembodiments described herein, it should be understood that in alternateimplementations of these embodiments, the target objects need not bewaste materials but may comprise and type of material for which it maybe desired to sort and/or segregate. Moreover, although a conveyer beltis used as an example conveyance mechanism for transporting the targetobjects within reach of the suction gripper, it should be understoodthat in alternate implementations of these embodiments, other conveyancemechanism may be employed. For example, for any of the embodimentsdescribed below, in place of an active conveyance mechanism such asconveyor belt, an alternate conveyance mechanism may comprise a chute,slide or other passive conveyance mechanism through and/or from whichmaterial tumbles, falls, or otherwise is gravity fed as it passes by theimaging device.

FIG. 1 is a diagram illustrating an optical material characterizationsystem 10 of one embodiment of the present disclosure. As shown in FIG.1, the optical material characterization system 10 comprises at leastone imaging device 162 directed at a conveyer belt 50 which transportsone or more target objects (shown at 55). In some embodiments theimaging device 162 comprises a camera sensor that captures visualspectrum images of the target objects 55 carried by the conveyor belt50. In other embodiments, the imaging device 162 may also, or instead,comprise infrared (IR) and/or ultraviolet (UV) sensors that capture IRand/or UV images of the target objects 55. As such, the imaging device162 may comprise visual spectrum, IR, or UV sensors, or some combinationthereof. It should also be appreciated that the imaging device 162 mayinstead, or further, comprise other types of sensors that may be used togenerate an image, such as but not limited to a magnetic sensor or ahyperspectral image sensor. In the embodiment shown in FIG. 1, theimaging device 162 functions as an area sensor that captures a twodimensional image (also referred to herein as an image “frame”) of anarea 60 of the conveyor belt 50, along with any objects 55 that arewithin that area 60. In other embodiments, the imaging device 162 maycomprise a line scanner that captures a strip or slice of the conveyorbelt 50 and its contents. These image strips may then be digitallyspliced to generate a composite image frame of area 60.

The imaging device 162 is in communication with an objectcharacterization processor 160 that comprises one or more NeuralProcessing Units 164 and a Neural Network Parameter Set 165 (whichstores learned parameters utilized by the Neural Processing Units 164).In some embodiments, the imaging device may also comprise a Data Storage166 that stores the raw images received from the imaging device 162,processed images comprising labeled data, and may further be used tostore other data such as material characterization data generated by theNeural Processing Units 164. The Neural Network Parameter Set 165 andData Storage 166 may either be implemented together on a common physicalnon-transient memory device, or on separate physical non-transientmemory devices. In some embodiments, the Data Storage 166 may comprise arandom access memory device, or removable storage media.

For some embodiments, the resulting material characterization dataoutput stored by the object characterization processor 160 (which insome embodiments may be stored to the data storage 166) may comprise atally of what different materials are present in the image frame. Insome embodiments, as further described below, the resulting materialcharacterization data may further provide how many objects of eachmaterial type are present in the image frame and/or the location of theidentified objects within the frame. For example, the materialcharacterization data may indicate that there are 3 aluminum cans withinthe image frame and provide coordinates for the geometric center of eachcan and/or an estimate of certain identified features such as the shapeand/or dimensions of the object. In some embodiments, the objectcharacterization processor 160 outputs one or more physical objectattributes determined by the one or more Neural Processing Units basedon visional inspection of the one or more target objects

The object characterization processor 160 as shown in FIG. 1 isimplemented as a discrete physical device comprising processingelectronics separate from those of the imaging device 162. In that case,the object characterization processor 160 would receive the imagesignals from the imaging device 162 via a wired or wirelesscommunication link. Alternatively, the imaging device 162 and objectcharacterization processor 160 may be integrated into a single physicalelectronics device and comprise the processors, memory and otherelectronics to implement the embodiments described herein. In variousembodiments, the object characterization processor 160 may beimplemented using a microprocessor coupled to a memory that is programedto execute code to carry out the functions of the objectcharacterization processor 160 described herein. In other embodiments,the object characterization processor 160 may additionally, oralternately, be implemented using an application specific integratedcircuit (ASIC) or field programmable gate array (FPGA) that has beenadapted for machine learning.

As shown in FIG. 1, the object characterization processor 160 mayfurther be coupled to at least one user interface device 170 (which maycomprise a display and/or user input interface such as a keyboard and/ortouchscreen, for example), a remote network 172 and/or process controlelectronics 174. In some embodiments, the user interface device 170provides a user access to the Data Storage 166 so that a user maymonitor images captured by the imaging device 162 and associatedmaterial characterization data generated by the object characterizationprocessor 160. Where Data Storage 166 comprises a removable storagemedia, that media may be inserted into a separate user terminal andsimilarly reviewed by a user. In some embodiments, the objectcharacterization processor 160 may transmit the contents of the datastorage 166 to one or more remote storage devices connected to theremote network 172 (via either wired or wireless network connections) orto a remote user interface device 170 (via either wired or wirelessnetwork connections) to provide access to the images and materialcharacterization data.

As also shown in FIG. 1, the optical material characterization system 10may comprise one or more light sources 176 controlled by the objectcharacterization processor 160 to illuminate the target objects 55within the area 60 of the conveyor belt 50 as the imaging device 162captures images of area 60 and its contents. In some embodiments, theobject characterization processor 160 selectively controls separatelight sources 176 that illuminate area 60 from different angles.Alternately, separate light sources 176 may selectively illuminate thearea 60 and its contents using different spectrums and/or wavelengths oflight, as further discussed below. In this way, the objectcharacterization processor 160 may obtain from imaging device 162multiple images of the same target object 55 under more than onelighting condition, which may provide data that can distinguishdifferent materials from each other, or be used to enhance estimates ofan object's size and dimensions, for example.

In operation the imaging device 162 is directed downward towards theconveyor belt 50 in order to capture an overhead view of the materials55 being transported by the conveyor belt 50. The imaging device 162produces an image signal that is delivered to the objectcharacterization processor 160. Within the object characterizationprocessor, these image frames are provided input to one or more neuralnetwork and artificial intelligence algorithms (shown as the NeuralProcessing Units 164) to locate and identify material appearing withinthe image frames.

Referring to FIG. 2, the Neural Processing Units 164 executed within theobject characterization processor 160 operate in two modes: 1) a machinelearning training mode (shown generally at 202), and 2) a machinelearning inference mode (shown generally at 204). Operating in themachine learning training mode, the Neural Processing Units 164 derivethe Neural Network Parameter Set 165 which is then used by the NeuralProcessing Units 164 in the machine learning inference mode to producethe material characterization data.

As shown in FIG. 2, a feed of image frames captured by the imagingdevice 162 (shown at 205) is stored into the data storage 166 (shown at210) to obtain a collection of image frames. For each of these storedimage frames, a user (for example, using user interface display 170 orremotely connecting to the object characterization processor 160 via theremote network 172) manually reviews the image (shown at 212) to producelabeled data (shown at 214). This labeled data 214 is stored back intothe data storage 166. To produce the labeled data 214, the user view acaptured image frame and via a user interface identifies where objects55 are present in the image fame (i.e., the objects location within theimage) and what the objects are. For example, the user may point to orcircle an object with the user interface and then tag the object withone of a predetermined set of labels based on what the user thinks theobject is. For example, the user may visually identify a first object ata first position within in the image as being a plastic bottle, andtherefore tag it with a “plastic bottle” label. The user may identify asecond object at a second position within in the image as an aluminumcan, and tag it with an “aluminum can” label. Further details regardingexamples of the process of labeling images which may be used in thepresent embodiments may be found in the reference “WHAT WE LEARNEDLABELING 1 MILLION IMAGES: A practical guide to image annotation forcomputer vision”, CrowdFlower Inc., 2017, which is incorporated byreference in its entirety. When the user has completed identifying andtagging the objects in the image frame, the labeled data 214 is savedback into the data storage 166 and associated with the raw image framereviewed by the user to create the labeled data 214.

Once the user can completed generating the labeled data 214 for a forthe set of raw image frames, both the labeled data 214 and raw imageframes are entered into a machine learning training algorithm 216 whichencodes the information (for example, by defining a set of pixels for anobject within the raw image frames as being the type material identifiedby the label with which it was tagged) and from that produce the NeuralNetwork Parameter Set 165 (shown at 218) that teaches the NeuralProcessing Units 164 how to identify material in subsequent images thatwill be provided when operating in machine learning inference mode. Inthe machine learning training mode the Neural Network Parameter Set 165is generated using the machine learning training algorithm 216 toproduce an optimized set of parameters. The purpose of these parametersis to configure the Neural Processing Units 164 to make the samedecision as the user made when producing the labeled data (though notnecessarily for the same reasons). One of ordinary skill in the artwould appreciate that there are a number of machine learning algorithmswhich may be utilized to implement the machine learning trainingalgorithm 216. Therefore, the machine learning training algorithm 216may comprise an algorithm, such as but not limited to StochasticGradient Descent (SGD), Adaptive Moment Estimation (ADAM), Root MeanSquare Propagation (RMSProp), Nesterov's First Order Method, theAdaptive Gradient Algorithm (AdaGrad), and/or variants thereof thatwould be known to those of ordinary skill in the art. For example, foradditional information regarding one of more of these algorithms whichmay be used in conjunction with the embodiments describer here, seeKingma et al., “ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION”, Proceedingsof the 3rd International Conference for Learning Representations, SanDiego, 2015, which is incorporated herein by reference in its entirety.

The output from the machine learning training algorithm 216 is theNeural Network Parameter Set 165 that allows the Neural Processing Units164 to identify material in the image frames. In some embodiments, theparameters that comprise the Neural Network Parameter Set 165 areassociated with various features of the imaged objects such as, but notlimited to hues, texture, grain, reflectance, printed graphics, edges,corners, and so forth. In one embodiment, the Neural Processing Units164 each comprise multiple processing layers (referred to a “neurons”)such that each processing layer searches for a particular feature fromthe data. For example, hue may be one feature considered by a processinglayer of the Neural Processing Units 164. If the target object has a huewithin a first range of hues that has a high correlation with the huesof a milk carton, then that features weights in favor for the objectbeing a milk carton. If the object in the image instead has a hue withina second range of hues that has a low correlation with the hues of amilk carton, then that features weights in favor of the object not beinga milk carton. Such correlations would be determined using neuralnetwork parameters defined for the particular feature as stored in theNeural Network Parameter Set 165. Other processing layers within theNeural Processing Units 164 would look for correlations for otherfeatures of the object in the image frame. However, it should be notedthat which features of the object are important to consider and/or workbest for differentiating objects are determined during training of theNeural Processing Units 164 with the machine learning training algorithm216.

Once the Neural Network Parameter Set 165 has been generated, the objectcharacterization processor 160 and Neural Processing Units 164 may beginoperation in the machine learning inference mode 204 in which the NeuralProcessing Units 164 is operating to detect materials within the imageframe rather than learn. It should be noted that once a Neural NetworkParameter Set 165 is generated, those parameters may be stored withinand used by the same object characterization processor 160 and NeuralProcessing Units 164 which created them. Alternatively, the NeuralNetwork Parameter Set 165 generated by one object characterizationprocessor 160 may optionally be distributed and loaded for use ontoanother object characterization processor 160 expected to perform thesame task as the object characterization processor 160 and NeuralProcessing Units 164 which created them under the same conditions.

During operation in machine learning inference mode (shown at 204), thefeed of image frames 205 captured by the imaging device 162 is feddirectly to the machine learning inference algorithm 220, where it issequentially processed by each of the multiple processing layers, orneurons, of the Neural Processing Units 164 to evaluate the correlationof specific features with features of objects that it has previouslylearned.

An example structure for one embodiment of a neural network of theNeural Processing Units 164 is illustrated generally at 300 in FIG. 3.As shown in FIG. 3 at 305, the image frame captured by the imagingdevice 162 into the first layer neuron 310-1. It should be appreciatedthat when an image frame is captured by the imaging device 162, aparticular object may only partially appear within the frame and/or maystraddle across consecutive frames. As such, imaging device 162 may beoperated to selectively capture either overlapping or non-overlappingframe sequences. For example, because the speed at which objects 55travel on the conveyor belt 50 is known and is typically constant, thenthe object characterization processor 160 can input two consecutiveframes and calculate how much of one frame overlaps with the next tonegate duplicative object data. Similarly, the object characterizationprocessor 160 may discard captured image frames until a frame iscaptured that contains only new data. If some materials have not yetcompletely entered area 60, then characterization processor 160 may waituntil those objects are entirely in-frame before processing an imageframe.

The first layer neuron 310-1 is configured using the Neural NetworkParameter Set 165 to detect a specific feature or combination offeatures. More specifically, the Neural Network Parameter Set 165comprises a subset of parameters learned during the machine learningtraining mode that are associated with this first layer neuron 310-1 forthe purpose of detecting this feature. The first layer neuron 310-1generates a plurality of neuron outputs 312-1 that characterize thepresence of that feature in the image frame 305. For example, if thefeature evaluated by the first layer neuron 310 is hue, then the neuronoutputs 312-1 may comprise outputs from multiple nodes of the firstlayer neuron 310-1, one output for each hue that the first layer neuron310-1 is trained to detect. If the feature evaluated by the first layerneuron 310 is hue, then nodes of the neuron outputs 312-1 that areassociated with any detected hues will go high. Nodes of the neuronoutputs 312-1 not associated with detected hues will go low.

The neuron outputs from the first layer neuron 310-1 are fed as inputsto the second layer neuron 310-2 which, in the same way, evaluates theimage frame for a different feature or combination of features. Theneuron outputs 312-2 from the second layer neuron 310-2 are fed asinputs to the next neuron to produce neuron outputs, and the layersrepeat so forth to neuron 310-n, each layer detecting a differentfeature or combination of features, and configured by a subset ofparameters learned during the machine learning training mode that areassociated with that layer neuron for the purpose of detecting thatfeature. The final neuron outputs 312-n from the output neuron 310-neach respectively represent a confidence level that a particularmaterial is present in the image frame 305. In other words, the finaloutput neuron 310-n will comprise an output node for each of thepossible materials that the Neural Processing Units 164 have learned,and output a value indicating a confidence level that the objectcomprises the material associated with that node. If an object in theimage comprises an aluminum can, then a node of the output neuron 310-nassociated with aluminum cans should provide an output with a highconfidence value. Meanwhile, the nodes of the output neuron 310-nassociated with other materials should provide an output with a lowconfidence value. In some embodiments, a confidence threshold isestablished to make the determination as to whether an object of acertain material is present in the image. This determination is shown at320 in FIG. 3. For example, if a confidence threshold for aluminum cansis set for 0.5 and the output node from neuron 310-n for aluminum cansoutputs a value of 0.3, then the determination at 320 would concludethat no aluminum can is present in the processed image frame 305.Conversely, if the output node from neuron 310-n for aluminum cansoutputs a value of 0.6, then the determination at 320 would concludethat at least one aluminum can is present in the image frame. In someembodiments, the same confidence threshold may be used for allmaterials. In other embodiments, the confidence threshold for differentmaterials may vary. For example, a confidence threshold used for silver,gold, or other precious (high value) materials may be set low (forexample 0.1) as compared to lesser value materials, in order to ensure agreater chance that such materials are identified as present and do notproceed into the system undetected.

The process described in FIG. 3 may generally be referred to by those ofskill in the art of machine learning as a “Classification” processbecause it processes the provided image frame in order to determine whatmaterial may be present in the image frame. In order to provide furtherdetails about the objects present in the image frame, such as the numberof objects comprising each detected material, and the location, shape,orientation, and dimensions of these objects in the image frame, a“Segmentation” or similar process known to those in the art of machinelearning may be used. Alternative algorithms to detect objects within animage include Fully Convolutional Neural Network, Multibox, Region-basedFully Convolutional Networks (R-FCN), Faster R-CNN, and other techniquescommonly known to those skilled in the art as object detection,instance-aware segmentation, or semantic segmentation algorithmsdescribed in available literature. Further details regarding examples ofthe process of detecting objects in captured images which may be used inthe present embodiments may be found in the reference Huang et al.,“Speed/accuracy trade-offs for modern convolutional object detectors”The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),2017, pp. 7310-7311, which is incorporated by reference in its entirety.

One such process is illustrated by the example embodiments shown in FIG.4. FIG. 4 illustrates generally at 400 a Fully Convolutional NeuralNetwork which may be referred to by those of skill in the art of machinelearning as a “Machine Learning Based Segmentation Algorithm.” In theprocess 400 shown in FIG. 4, as opposed to feeding the entirety of acaptured image frame into the neural network 300 of the NeuralProcessing Units 164, a cropped image frame 410 (referred to herein asan “image window”) comprising a portion of the full image frame capturedby imaging device 162 is fed as the input to the segmentation processshown in FIG. 3. Moreover, the neural network 300 is trained to providean output specifically indicating the material detected at a centerpixel 412 of the image window 410 that corresponds to a pixel atlocation (x,y) of the full image frame. In other words, when the imagewindow 410 is fed in as the input to the neural network 300, the neuronoutput 312-n for a particular material will go high only if thatmaterial is detected as overlapping the center pixel 412 of the imagewindow 410. In this way, the Neural Processing Units 164 canspecifically identify that an object comprising that specific materialis present at pixel location (x,y) of the full image frame. Then, byincrementally sweeping the position of the image window 410 over theentirety of the full image frame, feeding the image windows 410 at eachincrement into the neural network 300, each pixel location (x,y) of thefull image frame in which an object that comprises that specificmaterial is present can be identified to produce a composite materialspecific image (MSI) map 450 for that material. For example, assumingthat the MSI map 450 shown in FIG. 4 applies to aluminum cans, then asthe Neural Processing Units 164 detect that the material overlapping thecenter pixel 412 of the image window 410 is an aluminum can, then thecorresponding pixel location in the MSI map 450 may be set to high. Whenthe Neural Processing Units 164 detect that the material overlapping thecenter pixel 412 of the image window 410 is anything other than analuminum can, then the corresponding pixel location in the MSI map 450may be set to low. The resulting MSI map 450 will comprise a pattern ofpixels set high (shown at 452) where aluminum cans have been determinedto exists, further indicating their size an orientation. This sameprocess may be simultaneously performed for each of the differentmaterial that the Neural Processing Units 164 is trained to detect aseach image window 410 is processed so that multiple MSI maps 450 may begenerated in parallel (shown at 451). For example, if the NeuralProcessing Units 164 is trained to detect 50 distinct materials, then 50MSI maps 450 may be generated in parallel, each respective MSI map 450dedicate to illustrating where a particular material, and thus acorresponding object 55, is present in the full image frame. The MSImaps 450 represent material characterization data that may be storedinto the Data Storage 166 as described above, accessed by a user via auser interface display 170, transmitted to remote storage or a remoteuser interface display via a remote network 172, used to automaticallyactivate or control other facility machinery via process controlelectronics 174, and/or provided as input used by other facilitymachinery such as a sorting robot (such as described by any of theembodiments disclosed in U.S. Provisional Patent Application No.62/561,400, titled “SYSTEMS AND METHODS FOR ROBOTIC SUCTION GRIPPERS”and filed on Sep. 21, 2017, which is hereby incorporated by reference inits entirety) or air jet sorting mechanism or other sorting mechanism(shown at sorting mechanism 178). Although this specification describesspecific examples, it should be noted that any of the Segmentationtechniques known to those versed in the art may be used in alternatedembodiments to generate the MSI maps discussed herein.

In some embodiments, rather than an MSI map 450 depicting where aparticular material, and thus a corresponding object 55, is present inthe full image frame, the MSI map 450 may be further defined based on aregion of the object 55 that has a high correlation with a particularfeature evaluated by the Neural Processing Units 164. For example,sorting mechanism 178 may be provided one or more MSI maps 450 by objectcharacterization processor 160 in order to drive the sorting mechanism178 to capture and remove objects appearing in those MSI maps. In someembodiments, the MSI maps 450 provided by object characterizationprocessor 160 may be further refined to identify the specific surfaceareas of the identified objects having features that present the bestsurface for the gripper of the sorting mechanism 178 to attempt to grip.The gripper of the sorting mechanism 178 would have a higher probabilityof successfully capturing the object by gripping these surface areas ofthe identified objects as opposed to other areas. For example, where thesorting mechanism 178 comprises a suction gripper that it utilized tocapture and remove target objects, the MSI maps 450 may highlight thosepixel coordinates corresponding to regions having characteristics mosteasily captured using a suction gripper (for example, region of theobject comprising a flat surface, or at least the flattest surface). Inone embodiment, instead of an MSI map 450 for a given material havingpixel set to high based on the Neural Processing Units 164 detectingthat the material overlapping the center pixel 412 of the image window410 comprises that given material, it would instead be set to high whenthe material overlapping the center pixel 412 of the image window 410comprises that given material and presents the best surface on thetarget object for the sorting mechanism 178 to attempt to grasp tocapture the target object. When the sorting mechanism 178 receives thatMSI map 450, it will thus know not only which objects on the conveyorbelt should be picked and removed, but the best location on the objectfor it to attempt to grasp using its gripper.

As mentioned above, illuminating target objects 55 using a diversity oflight sources from different angles or using different spectrum mayprovide additional data which may be used to better differentiatedifferent materials that may appear similar under standard lightingconditions.

As such, in some embodiments the object characterization processor 160controls the light sources 176 so that imaging device 162 capturesmultiple image frames of the objects 55 in area 60 in succession underdifferent lighting conditions. For example, two or more light sources176 may be flashed from different angles to generate images withdifferent shadows for the same objects to provide contextual informationthat is used, for example, to determine the dimensions of a singleobject, or distinguish separate objects from each other. Alternately,the light sources 176 may illuminate the objects 55 using lightcomprising different wavelengths while imaging device 162 capturesmultiple image frames of the objects 55 in area 60 in succession. Forexample, the light sources 176 may switch between visible spectrumfrequencies (such as red, blue, yellow). Further, the light sources 176may switch between a first spectrum of visible light and a secondspectrum of non-visible light. For example, some materials may emit afluorescence under UV lighting that may distinguish them from othermaterials that look the same under visible light.

In such embodiments, the collection images frames for the set of objects55 within area 60 under the different lighting conditions are consideredcollectively as the unit of data under which the Neural Processing Units164 are trained, and the same lighting conditions are applied during theoperation in machine learning inference mode. That is, the multiple rawimage frames together with the associated labeled data generated intraining mode would be entered as a single unit of data into the machinelearning training algorithm for generation of the Neural NetworkParameter Set 165. Moreover, during machine learning inference mode theobject characterization processor 160 may selectively controls the lightsources 176 based on input from a sensor, or based on a determinationmay be the Neural Processing Units 164. For example, the objectcharacterization processor 160 may control the light sources 176 toilluminate the objects 55 in area 60 under a visible light spectrumunder for steady state operation, but if the Neural Processing Units 164detects the presence of a certain material, it then temporarily switchesto a different second spectrum to obtain an additional image frameunder. The additional image frame captured under the second spectrum maythen be evaluated by the Neural Processing Units 164 to either confirmthat the initial material determination was correct, or to morespecifically differentiate and identify the material. In someembodiments such as described above, the Neural Processing Units 164 maycomprise a first neural network to evaluate images captured under afirst set of lighting conditions, and one or more additional neuralnetwork to evaluate images captured under respective additional lightingconditions.

Besides the control of lighting based on determinations made by theNeural Processing Units 164, the object characterization processor 160may also control other automatic and/or real time processes based on thematerial characterization data. For example, the object characterizationprocessor 160 may be coupled to process control electronics 174, whichmay comprise, for example, one or more actuators, servos, processcontrollers, lighting controls, and the like. For example, in someembodiments, the object characterization processor 160 may send acontrol signal to process control electronics 174 to adjust theoperating speed of the conveyor belt 50 and/or actuate other machineryin the facility, to provide the location of a specific target object 55to a sorting mechanism 178 to pick the material from the conveyor belt50, or to activate alarms or warning indicators (for example, if an itemof hazardous material has been detected). For example, in oneembodiment, based on the material characterization data, the objectcharacterization processor 160 recognizes that an object 55 comprising apredetermined material has been detected within the area 60. The objectcharacterization processor 160 can then send material characterizationdata that comprises the coordinate information for the target object 55to the sorting mechanism 178 to remove that object from the conveyorbelt as described above. In other embodiments, the objectcharacterization processor 160 may detect that too much, or not enoughof, a specific material is being detected and operate process controlelectronics 174 to operate other machinery (such as other conveyorbelts, or the operation of disc screen sorting devices) to increase ordecrease the amount of those materials it is detecting.

FIG. 5 is a diagram illustrating generally at 500 an example display ofmaterial characterization data at a user interface device for oneembodiment of the present disclosure. This display is intended toprovide only an example of the manner in which material characterizationdata, and statistics and other information derived from the materialcharacterization data by the object characterization processor 160, maybe generated and displayed. As such, FIG. 5 is not intended to be alimiting example. For example, at 510, statistics showing the cumulativetotal number of materials detected over a given period, and at 511 theestimated mass of those materials, may calculated by the objectcharacterization processor 160. Estimated mass may be calculated, forexample, by correlating the number of objects of each material typeobserved against a look-up table stored in the memory of objectcharacterization processor 160 that provides a standard mass estimatefor an item of that material. Moreover, the memory of objectcharacterization processor 160 may be updated to include current marketvalues for one or more of the materials detected so that the marketvalue of that material collected by the facility and detected by theoptical material characterization system 10 may be provided. In someembodiments, the displayed may be filtered to provide such statisticsfor specific materials. For example, the user may request the materialcharacterization data for just aluminum cans over the past 24 hours. Inthat case, the display 500 may show information such as the total numberof aluminum cans detected, an estimate of the total mass of aluminumcans detected, and the current market value for the aluminum cansdetected. In some embodiments, material characterization data may begraphically displayed. For example, graphical data (shown at 515) mayincrementally display the number of times a specific material has beendetected over a previous time period. The object characterizationprocessor 160 may also process the material characterization data todisplay statistics on hazardous material collected (shown at 520) ormaterials considered to be contaminant materials (shown at 525). In someembodiments, a live video feed of image frames from the imaging device162 may be displayed (shown at 530).

Although the embodiments above each describe the optical materialcharacterization system 10 operating in the context of a recyclingand/or waste management facility, it should be appreciated thatembodiments operating within the context of other facilities isexpressly contemplated. For example, in one embodiment, the opticalmaterial characterization system 10 is implemented within the context ofa facility that operates one or more anaerobic digesters. As would beappreciated by those skilled in the art, anaerobic digestion comprises aseries of biological processes in which microorganisms break downbiodegradable material in the absence of oxygen. The end product of thisprocess may be used to produce biofuels, such as biogas, which may becombusted, for example to generate electricity and/or heat, or processedinto renewable natural gas and transportation fuels. In one embodimentof the optical material characterization system 10, the conveyor belt 50transports objects 55, which in this embodiment comprise food waste thatserves as fuel to feed one or more anaerobic digestion vats in whichanaerobic digestion takes place. In one such embodiment, the conveyorbelt 50 may carry the food waste directly into the intake of ananaerobic digestion vat. The imaging device 162 captures images of thevarious objects 55 comprising food waste within area 60 and the objectcharacterization processor 160 may be configured to detect and identifyexactly what types of food waste will be entering the anaerobicdigestion vat. Accordingly, the Neural Processing Units 164 will betrained and operated in both the machine learning training mode andmachine learning inference mode described above in the same matter asdescribed above to identify different materials. However, instead ofhaving the neural network parameter set 165 trained to detect anddistinguish recycled materials such as aluminum cans, cardboard, andmilk jugs, it is instead trained to detect and distinguish differentfood waste materials. The object characterization processor 160 can thencalculate and output an indication of the total available energy (forexample, available BTU) that will be supplied to the anaerobic digestionvat by the food waste materials in that image frame. For example, theNeural Processing Units 164 may be trained to identify food wastematerials such as banana peels, apple cores, or indeterminate food wastematerial such as generic brown sludge. Using the processes describedabove for operation in machine learning inference mode above, the NeuralProcessing Units 164 may analyze a captured image frame and identify thetotal mass respectively for each of the banana peels, apple cores, andgeneric brown sludge present on that section of the conveyor belt andreference data available in memory (e.g. a look-up table stored in thememory of object characterization processor 160) to retrieve BTU/unitmass multipliers for each of these materials. The total available BTU isthen obtained by multiplying the detected mass for each food wastematerial by its corresponding multiplier and summing the results. Insome embodiments, based on the determined total BTU, the objectcharacterization processor 160 may output control signals (e.g. viaprocess control electronics 174) to vary the operation of other facilityequipment. For example, the based on the BTU availability provided bythe food waste detected by the Neural Processing Units 164, the objectcharacterization processor 160 may adjust the speed of the conveyor belt50 to maintain, or converge on, a target BTU input rate. In otherimplementations, other external devices and process flows for operatingthe anaerobic digestion vat may be controlled based on the BTUavailability and/or materials detected by the Neural Processing Units164. For example, in one embodiment, if a contaminant is detected thatwould adversely affect operation of the anaerobic digestion vat, thatmaterial may be identified and diverted off of the conveyor feeding theanaerobic digestion vat by a sorting robot or other deice such asdescribed above. In other embodiments, the conveyor belt 50 may notcarry the food waste directly into the intake of an anaerobic digestionvat, but instead load storage silos that are subsequently used to feedthe anaerobic digestion vat. In such an embodiments, the BTUavailability provided by the food waste detected by the NeuralProcessing Units 164 may be used to regulate the loading of food wasteinto the silos based on the BTU of materials loaded rather than mass orweight. In one embodiment, once the target BTU for a silo is obtained,the object characterization processor 160 may provide a signalindicating such to plant equipment to realign the conveyor belt 50 tofeed another silo. In this way, a more accurate measurement of the BTUavailable from a particular silo can be established. It should thereforebe appreciated that in still other embodiments, characterization ofconveyed objects for still other applications are expresslycontemplated.

FIG. 6 is a diagram illustrating generally at 600 another exampleimplementation of the optical material characterization system 10discussed above (which will now be referred to as optical materialcharacterization system 610) adapted for use outside of a fixedfacility, and instead implemented in a moving collection vehicle 605such as, but not limited to, a waste collection truck. It should beunderstood that the features and elements described herein with respectto FIG. 1 may be used in conjunction with, in combination with, orsubstituted for elements of FIG. 6 and optical material characterizationsystem 610, as well as any of the other embodiments discussed herein,and vice versa. Further, it should be understood that the functions,structures and other description of elements for embodiments describedherein may apply to like named or described elements for any of theFigures and vice versa.

In some embodiments, object characterization processor 160, imagingdevice 162, one or more light sources 176, and other components ofsystem 610 may be located onboard the collection vehicle 605 andfunction in the same manner as described above. In other embodiments,only a subset of the components of system 610 are located onboard thecollection vehicle 605. For example, in one embodiment, the imagingdevice 162 and one or more optional light sources 176 may be locatedonboard the collection vehicle 605 while the object characterizationprocessor 160 is located at a central station and is in wirelesscommunication with the imaging device 162 and one or more light sources176 such as through cellular or Wi-Fi, or other wireless communicationstechnology. In that case, raw image frames from the imaging device 162may be wirelessly transmitted to the central station for subsequentprocessing by neural processing units 164. In some embodiments, the rawimage frames may instead be locally stored on a non-transient memorydevice (for example, such as removable memory card or disk) aboard thecollection vehicle 605 and transferred to the object characterizationprocessor 160 when the collection vehicle 605 arrived as the centralstation. In some embodiments, the optical material characterizationsystem 610 may further comprise locally at the collection vehicle 605 atleast one user interface device 170 and/or process control electronics174. For example, in one embodiment in operation, the Collection Vehicle605 includes a mechanical loading arm 630 with which it lifts acollection bin 632 in order to deposit the contents of the collectionbin 632 into the collection vehicle 605. In one embodiment the loadingarm 630 transports materials to be collected to a part of the collectionvehicle, which may comprise an active or passive conveyance mechanism.In other embodiments, the collection bin 632 may be deposited into thevehicle 605 manually without the aid of a mechanical loading arm 630.

If as (and/or after) the materials enter the collection vehicle 605 theobject characterization processor 160 detects an anomaly such ashazardous or other prohibited material, the optical materialcharacterization system 610 may issue a warning or other form ofspecific instructions to the operator of the collection vehicle 605 viathe onboard user interface device 170. Such a warning and/or otherspecific instructions may be tailored to the specific materialidentified by the system 610. Moreover, the process control electronics174 may be used to control Mechanical Loading Arm 630 (or otherwiseoverride local operation). For example, where object characterizationprocessor 160 detects undesirable materials, it may send a controlsignal to the process control electronics 174, which in turn deactivatescontinued operation of the Mechanical Loading Arm 630 instead of, or inaddition to, outputting a message to the onboard user interface device170.

FIG. 7 illustrates at 700 an example embodiment of optical materialcharacterization system 610 in operation where the collection vehicle605 receives material from collection bin 632 from above. That is, thecollected material 655 (which would correspond to the target object 55discussed above) can be fed (for example, gravity fed) into thecollection vehicle 605. In some embodiments, the materials 655 mayoptionally travel down a conveyance mechanism 650 such as a slide orchute which may further function to spread materials out as they enterthe vehicle. Upon entry into the collection vehicle 605, the collectedmaterial 655 pass through the field of view of imaging device 162 andare characterized in the same manner described above for system 10. Asdescribed above, the imaging device 162 is directed towards thecollected material 655 entering the collection vehicle 605 produces animage signal that is delivered to the object characterization processor160. Within the object characterization processor, these image framesare provided input to one or more neural network and artificialintelligence algorithms (shown as the Neural Processing Units 164) tolocate and identify material appearing within the image frames. FIG. 7Aillustrates at 750 an another example embodiment of optical materialcharacterization system 610 in operation where the collection vehicle605 is back or side loaded with material from collection bin 632. Theimaging device 162 is directed towards the collected material 655entering the collection vehicle 605 and produces an image signal that isdelivered to the object characterization processor 160. Within theobject characterization processor, these image frames are provided inputto one or more neural network and artificial intelligence algorithms(shown as the Neural Processing Units 164) to locate and identifymaterial appearing within the image frames.

In either of the implementations shown in FIG. 7 or 7A, newly collectedmaterial 655 will tend to be deposited over previously collectedmaterial 655. As such, as shown in FIG. 8, image frames captured by theimaging device 162 of collected material 655 within the body ofcollection vehicle 605 (shown at 810) may include a combination of newlycollected material 655 and previously collected material 655. Theseimage frames 810 may be fed into the Machine learning inferencealgorithm 220 of the Neural Processing Units 164 (shown at 820) in orderto detect, for example, object material type, location, shape,orientation and size (shown at 830) of the collected material 655appearing in each frame, in the same manner as described above forsystem 10. In one embodiment, this information is then processed by theoptical material characterization system 610 to determine which of thoseobjects were previously present within the collection vehicle 605, andwhich are newly present (shown at 840). For example, if there is a highcorrelation of characteristics (for example object material type,location, shape, orientation and/or size) between an item of collectedmaterial 655 appearing in the image frame 810 and a previouslyidentified object from a previously image frame, then that item ofcollected material 655 is not considered to be a newly collected object(shown at 850). If there is no such correlation, the item of collectedmaterial 655 is considered to be a newly collected object. In oneembodiment, the material characterization system 610 comprises a memory844 (for example, Data Storage 166) where a list or catalog of objectscollected and characterized by optical material characterization system610 are stored. Once newly collected objects 655 are characterized, theyare added to the list or catalog of objects collected (shown at 842) andfrom that point forward are considered previously collected objects 655.

FIG. 9 is a diagram illustrating generally at 900 another exampledisplay of material characterization data at a user interface device forone embodiment of the present disclosure. In one embodiment, thisexample display of material characterization data is used in combinationwith the optical material characterization system 610 and may bedisplayed at a local user interface device 170 located aboard collectionvehicle 605, or at a user interface device 170 located remote fromcollection vehicle 605, such as at a central station, for example. Asshown at 910, image frames captured by the optical materialcharacterization system 610 may be further tagged with locationinformation (shown at 910) and time and date information (shown at 920)so that collected objects 655 may be correlated to where and when theywere collected. In some embodiments, when am object 655 considered to bea contaminant is collected, such information may be displayed on theuser interface device 170 along with optional images of the offendingobjects (shown at 920 and 921). A count of the number of collectedobjects considered contaminants may also be displayed (as shown at 912).This information may allow the vehicle operator to locate thecontaminant for removal or other mitigation. Moreover, this informationmay be used to fine customers that place contaminants into thecollection bin, or alternatively provide rebates to customers if theyrecycle “high value” materials. In some embodiments, a live video feedof image frames captured by imaging device 162 may be displayed (shownat 922).

Example Embodiments

Example 1 includes an optical material characterization system, thesystem comprising: at least one imaging device configured to generateimage frames that capture an image of an area and one or more targetobjects within the area; an object characterization processor coupled tothe at least one imaging device and comprising one or more NeuralProcessing Units and a Neural Network Parameter Set, wherein the NeuralNetwork Parameter Set stores learned parameters utilized by the one ormore Neural Processing Units for characterizing the one or more targetobjects; wherein the one or more Neural Processing Units are configuredby the Neural Network Parameter Set to detect a presence of a pluralityof different materials within the image frames based on a plurality ofdifferent features; and wherein for a first image frame of the pluralityof image frames, the one or more Neural Processing Units outputsmaterial characterization data that identifies which of the plurality ofdifferent materials are detected in the first image frame.

Example 2 includes the system of example 1, wherein the at least oneimaging device is positioned to generate image frames that capture animage of an area of a conveyance mechanism and the one or more targetobjects are transported by the conveyance mechanism positioned withinthe area.

Example 3 includes the system of example 2, wherein the conveyancemechanism comprises a conveyor belt, a slide, a chute, an openingthrough which the one or more target objects are dropped, or a mechanismthrough which one or more target objects are gravity fed.

Example 4 includes the system of any of examples 2-3, wherein one orboth of the at least one imaging device and the conveyance mechanism aremounted within a vehicle and a loading arm transports materials to theconveyance mechanism.

Example 5 includes the system of any of examples 1-4, wherein the areais located within a collection vehicle.

Example 6 includes the system of example 5, wherein the at least oneimaging device is located within the collection vehicle.

Example 7 includes the system of any of examples 5-6, wherein the areaincludes a part of a collection vehicle that receives the plurality ofdifferent materials, and the at least one imaging device is configuredto generate image frames of the plurality of different materialsreceived by the part of the collection vehicle.

Example 8 includes the system of any of examples 1-8, wherein thematerial characterization data further comprises at least one of alocation, a shape, a size, and an orientation of at least a firstmaterial of the plurality of different materials detected in the firstimage frame.

Example 9 includes the system of example 8, wherein the one or moreNeural Processing Units produce a material specific image (MSI) mapillustrating a position for at least the first material within the firstimage frame.

Example 10 includes the system of example 9, wherein the objectcharacterization processor outputs the MSI map to a sorting robot toidentify locations of one or more of the target objects for the sortingrobot to capture.

Example 11 includes the system of example 10, wherein the MSI mapidentifies locations of specific surface areas of the one or more of thetarget objects having features that present a surface most suitable forgripping by the sorting robot based on features detected by the one ormore Neural Processing Units.

Example 12 includes the system of any of examples 1-11, wherein theobject characterization processor outputs one or more physical objectattributes determined by the one or more Neural Processing Units basedon visional inspection of the one or more target objects.

Example 13 includes the system of any of examples 1-12, wherein based onwhich of the plurality of different materials are detected in the firstimage frame, the object characterization processor outputs a controlsignal to control operation of a device external to the optical materialcharacterization system.

Example 14 includes the system of any of examples 1-13, furthercomprising a Data Storage that stores the image frames received from theimaging device

Example 15 includes the system of any of examples 1-14, wherein thematerial characterization data is either saved to a Data Storage, orcommunicated to a remote device over a remote network.

Example 16 includes the system of any of examples 1-15, furthercomprising a user interface device coupled to the objectcharacterization processor, wherein object characterization processor isconfigured to display the material characterization data, and statisticsand other information derived from the material characterization datavia the user interface device.

Example 17 includes the system of example 16, wherein the user interfacedevice is communicatively coupled to the object characterizationprocessor via a network.

Example 18 includes the system of any of examples 1-17, wherein based onthe material characterization data, the object characterizationprocessor calculates at least one of a total mass, a total market value,or a total available energy, for a least one of the plurality ofdifferent materials detected in the first image frame.

Example 19 includes the system of any of examples 1-18, wherein the atleast one imaging device comprises at least one of a visual spectrumimage sensor, a hyperspectral image sensor, an infrared image sensor, anultraviolet image sensor, or a magnetic image sensor.

Example 20 includes the system of any of examples 1-19, wherein theobject characterization processor is integrated within the at least oneimaging device.

Example 21 includes the system of any of examples 1-20, wherein theobject characterization processor is coupled to receive the image framesfrom the at least one imaging device by either a wired or wirelesscommunication link.

Example 22 includes the system of any of examples 1-21, furthercomprising one or more light sources, wherein the one or more lightsources are controlled by the object characterization processor toilluminate the target objects within the as the imaging device generatesimage frames.

Example 23 includes the system of example 22, wherein the one or morelight sources are selectively controlled to illuminate the targetobjects from different angles or using different wavelengths of lights.

Example 24 includes the system of example 23, wherein the one or morelight sources provide at least one of visual spectrum, infrared, orultraviolet light.

Example 25 includes the system of any of examples 23-24, wherein the oneor more light sources are selectively controlled by objectcharacterization processor based on materials detected in the area bythe one or more Neural Processing Units.

Example 26 includes an object characterization processing system, thesystem comprising: one or more Neural Processing Units; a Neural NetworkParameter Set; wherein the one or more Neural Processing Units receive,from at least one imaging device, image frames each comprising an imageof an area; wherein the Neural Network Parameter Set stores learnedparameters utilized by the one or more Neural Processing Units forcharacterizing one or more target objects appearing in the image frames;wherein the one or more Neural Processing Units are configured by theNeural Network Parameter Set to detect a presence of a plurality ofdifferent materials within the image frames based on a plurality ofdifferent features; wherein based on which of the plurality of differentmaterials are detected in a first image frame by the one or more NeuralProcessing Units, the object characterization processing system outputsa signal.

Example 27 includes the system of example 26, wherein the area islocated within a collection vehicle.

Example 28 includes the system of example 27, wherein the at least oneimaging device is located within the collection vehicle.

Example 29 includes the system of any of examples 27-28, wherein thearea includes a part of a collection vehicle that receives the pluralityof different materials, and the at least one imaging device isconfigured to generate image frames of the plurality of differentmaterials received by the part of the collection vehicle.

Example 30 includes the system of any of examples 26-29, wherein thesignal comprises data describing which of the plurality of differentmaterials are detected in the first image frame and the data is storedto a data storage.

Example 31 includes the system of any of examples 26-30, wherein thesignal causes a user interface device to output a display of statisticson material collected.

Example 32 includes the system of any of examples 26-31, wherein thesignal causes a user interface device to output a display of a materialspecific image (MSI) map illustrating a position for where at least afirst material of the plurality of different materials are detected.

Example 33 includes the system of any of examples 26-32, wherein basedon which of the plurality of different materials are detected in a firstimage frame by the one or more Neural Processing Units, the objectcharacterization processing system outputs a control signal to controloperation of a device external to the optical material characterizationsystem.

Example 34 includes an object characterization method, the methodcomprising: receiving images frames from at least one imaging device,the image frames each comprising an image of an area; evaluating theimage frames using one or more Neural Processing Units to detect apresence of a plurality of different materials within the image framesbased on a plurality of different features, wherein the one or moreNeural Processing Units are configured by a Neural Network Parameter Setto characterize one or more objects appearing in the image frames; andbased on which of the plurality of different materials are detected in afirst image frame of the image frames by the one or more NeuralProcessing Units, outputting a signal.

Example 35 includes the method of example 34, wherein the area islocated within a collection vehicle.

Example 36 includes the method of example 35, wherein the at least oneimaging device is located within the collection vehicle.

Example 37 includes the method of any of examples 35-36, wherein thearea includes a part of a collection vehicle that receives the pluralityof different materials, and the at least one imaging device isconfigured to generate image frames of the plurality of differentmaterials received by the part of the collection vehicle.

Example 38 includes the method of any of examples 34-37, wherein thesignal comprises data describing which of the plurality of differentmaterials are detected in the first image frame and the data is storedto a data storage.

Example 39 includes the method of any of examples 34-38, wherein thesignal causes a user interface device to output a display of statisticson material collected.

Example 40 includes the method of any of examples 34-39, whereinoutputting the signal comprises outputting a control signal to controloperation of a device.

Example 41 includes the method of example 40, wherein outputting thesignal to control operation of the device comprises adjusting a speed ofa conveyance mechanism.

Example 42 includes the method of any of examples 40-41, whereinoutputting the signal to control operation of the device comprisesoutputting a material specific image (MSI) map to identify to the devicelocations of one or more of the objects for the device to capture.

Example 43 includes the method of any of examples 40-42, whereinoutputting the signal to control operation of the device comprisesselectively controlling one or more light sources based oncharacteristics of materials detected in the area by the one or moreNeural Processing Units.

In various alternative embodiments, system elements, method steps, orexamples described throughout this disclosure (such as the objectcharacterization processor, neural processing units, process controlelectronics, sorting robots and/or sub-parts of any thereof, forexample) may be implemented using one or more computer systems, fieldprogrammable gate arrays (FPGAs), or similar devices and/or comprising aprocessor coupled to a memory and executing code to realize thoseelements, processes, steps or examples, said code stored on anon-transient data storage device. Therefore other embodiments of thepresent disclosure may include elements comprising program instructionsresident on computer readable media which when implemented by suchcomputer systems, enable them to implement the embodiments describedherein. As used herein, the term “computer readable media” refers totangible memory storage devices having non-transient physical forms.Such non-transient physical forms may include computer memory devices,such as but not limited to punch cards, magnetic disk or tape, anyoptical data storage system, flash read only memory (ROM), non-volatileROM, programmable ROM (PROM), erasable-programmable ROM (E-PROM), randomaccess memory (RAM), or any other form of permanent, semi-permanent, ortemporary memory storage system or device having a physical, tangibleform. Program instructions include, but are not limited tocomputer-executable instructions executed by computer system processorsand hardware description languages such as Very High Speed IntegratedCircuit (VHSIC) Hardware Description Language (VHDL).

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement, which is calculated to achieve the same purpose,may be substituted for the specific embodiment shown. This applicationis intended to cover any adaptations or variations of the presentedembodiments. Therefore, it is manifestly intended that embodiments belimited only by the claims and the equivalents thereof.

What is claimed is:
 1. A moveable system, comprising: a loadingmechanism configured to deposit a set of objects into the moveablesystem; and an imaging device configured to: generate an image frame ofat least a subset of the set of objects within the moveable system; andcause a processor to identify a target object within the image frame. 2.The moveable system of claim 1, wherein the processor is located localto the moveable system.
 3. The moveable system of claim 1, wherein theprocessor is located in a central station and wherein the image frame issent from the moveable system to the processor over wirelesscommunications.
 4. The moveable system of claim 1, wherein the processoris located in a central station and wherein the image frame is sent fromthe moveable system to the processor over a wired communication link. 5.The moveable system of claim 1, further comprising a user interfacedevice, wherein an anomaly is detected by the processor based at leastin part on the image frame, and wherein the user interface device isconfigured to present a warning based at least in part on the detectionof the anomaly.
 6. The moveable system of claim 1, wherein anundesirable material is detected by the processor based at least in parton the image frame, and wherein the loading mechanism is configured toreceive a control signal to deactivate operation based at least in parton the detected undesirable material.
 7. The moveable system of claim 1,further comprising a conveyance mechanism, wherein the set of objects isdeposited onto the conveyance mechanism.
 8. The moveable system of claim1, wherein the processor is configured to identify the target objectwithin the image frame by inputting the image frame into a neuralnetwork or an artificial intelligence technique that is configured tolocate and identify material appearing within the image frame.
 9. Themoveable system of claim 1, wherein a first object that is previouslypresent in the moveable system and a second object that is newly presentin the moveable system are determined by the processor based at least inpart on the image frame.
 10. The moveable system of claim 9, wherein thesecond object is determined to be newly present in the moveable systemby comparing the second object against a list or catalog of previouslycollected objects.
 11. The moveable system of claim 9, wherein thesecond object that is newly present in the moveable system is added to alist or catalog of previously collected objects.
 12. The moveable systemof claim 1, further comprising a user interface device, wherein the userinterface device is configured to present the image frame.
 13. Themoveable system of claim 1, further comprising a user interface device,wherein the user interface device is configured to present the imageframe with location information associated with a location at which theset of objects was collected.
 14. The moveable system of claim 1,further comprising a user interface device, wherein the user interfacedevice is configured to present the image frame with an indication of acontaminant being detected within the image frame.
 15. The moveablesystem of claim 1, further comprising one or more light sources toilluminate the set of objects.
 16. A method, comprising: depositing,using a loading mechanism, a set of objects into a moveable system;generating an image frame of at least a subset of the set of objectswithin the moveable system; and causing a processor to identify a targetobject within the image frame.
 17. The method of claim 16, wherein theprocessor is located local to the moveable system.
 18. The method ofclaim 16, wherein the processor is located in a central station andwherein the image frame is sent from the moveable system to theprocessor over wireless communications.
 19. The method of claim 16,wherein an anomaly is detected by the processor based at least in parton the image frame, and the method further comprising presenting awarning based at least in part on the detection of the anomaly.
 20. Themethod of claim 16, wherein the processor is configured to identify thetarget object within the image frame by inputting the image frame into aneural network or an artificial intelligence technique that isconfigured to locate and identify material appearing within the imageframe.